fix(service-ai): estimate non-zero token usage in MemoryLLMAdapter#2000
Merged
Conversation
The in-memory adapter stands in for a real LLM in dev/CI/local-E2E, but
returned a flat `usage: {0,0,0}` on every path. That made every
token-metering feature — quota guardrails, lifetime caps, usage
dashboards, cost stops — impossible to exercise without a paid provider
key, i.e. without spending real money.
Estimate usage crudely (~4 chars/token, the standard ballpark) from the
prompt messages and echoed output across chat/complete/streamChat/
generateObject. Intentionally NOT provider-accurate — it exists so
usage-driven behaviour is testable money-free, not to bill anyone.
Adds a unit test asserting non-zero, additive (total = prompt+completion),
monotonic-in-input-size usage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📓 Docs Drift CheckThis PR changes 1 package(s): 5 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MemoryLLMAdapteris the money-free stand-in for a real LLM in dev, CI, and local E2E. But it hardcodedusage: { promptTokens: 0, completionTokens: 0, totalTokens: 0 }on every return path (chat,complete,streamChatfinish,generateObject, tool-call and error branches).Consequence: any feature that meters tokens — cloud's AI quota guardrail, free-tier lifetime caps, paid monthly hard cap, usage dashboards, cost self-stop — could not be exercised end-to-end without a paid provider key. You literally had to spend real API money to test a cost-control feature.
Fix
Estimate usage crudely from the prompt messages + echoed output (~4 chars/token, the standard ballpark) and return it on all paths via two small helpers (
estimateTokens/estimateUsage).This is intentionally not provider-accurate — it only needs to be non-zero, additive (
total = prompt + completion), and grow with input size, so usage-driven behaviour is testable money-free.Test
Adds a unit test asserting the new usage is non-zero, additive, and monotonic in input size. Full file: 103/103 pass.
Verification
Confirmed live in local E2E: assistant
ai_messagesrows now meter ~5000 tokens each (previously 0), which lets the cloud token guardrail actually accumulate toward its caps using the echo adapter.🤖 Generated with Claude Code